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1.  Objective 


Robotic  swarms  are  arrays  of  small  robots  capable  of  autonomous  travel  and  operation  as  a  unit 
on  land,  sea,  and  air.  In  theory,  the  effective  employment  of  robotic  swarms  can  accomplish 
critical  Army  tasks,  such  as  accompanying  convoys,  mapping  battlefields,  and  clearing 
minefields.  However,  swarm  control  may  become  unstable  when  disturbed  by  unexpected 
changes  in  weather  or  terrain,  degradation,  attrition,  or  enemy  actions,  any  of  which  could 
negatively  impact  or  tenninate  the  swarm’s  mission.  A  critical  aspect  of  swann  control, 
especially  in  disrupted  or  degraded  conditions,  is  Soldier-swarm  interaction.  The  Soldier  must 
remain  cognizant  of  swarm  operations,  which  is  achieved  through  an  interface  that  is  used  to 
monitor  status  and  institute  corrective  actions  during  swann  supervision  or  over-watch.  The 
growing  body  of  human-robot  interaction  (HRI)  research  still  has  little  to  say  about  the  design  of 
Soldier-swarm  interface  displays  and  controls. 

The  objective  of  this  two-year  Director’s  Research  Initiative  (DRI)  research  plan  was  to  design 
algorithms  and  devices  that  allowed  Soldiers  to  efficiently  interact  with  a  robotic  swarm 
participating  in  a  representative  convoy  mission.  The  year  one  objectives  were  to  (1)  develop 
metacognition  algorithms  that  enable  swann  members  to  efficiently  monitor  changes  in  swarm 
status  as  they  execute  the  mission  and  (2)  provide  display  concepts  that  can  efficiently  and 
effectively  communicate  swann  status  to  Soldiers  in  challenging  battlefield  environments. 


2.  Approach 


To  realize  their  potential,  robotic  swanns  must  operate  efficiently  among  members  within  the 
swarm  as  well  as  with  external  factors  such  as  a  Soldier/operator  in  a  vehicle.  Our  approach 
consisted  of  defining  swann  and  mission  characteristics,  defining  swarm  control,  providing 
metacognition  to  swarm  members,  designing  a  human-swann  interface,  and  then  conducting  a 
laboratory  study  to  evaluate  the  swarm  behavior  algorithms  and  the  Soldier-swarm  display. 

In  defining  swann  and  mission  characteristics,  we  used  a  simulated  swann  because  limitations  of 
technology,  budget,  and  time  resources  would  not  allow  us  to  use  a  swann  with  actual  robots. 
However,  using  a  swarm  simulation  had  the  added  benefits  of  allowing  the  analysis  of  swann 
size  (number  of  members)  and  type  (ground,  air  or  micro  systems),  as  well  as  the  examination  of 
different  metacognition  and  Soldier-swarm  interface  technologies.  We  concentrated  on  swarm 
missions  that  require  human  oversight  or  supervision.  In  particular,  we  focused  on  a  swann  that 
accompanies  a  manned  convoy  traveling  on  a  road  network  to  limit  the  number  of  simultaneous 
swarm  activities  to  a  small  and  predictable  number.  In  our  final  approach,  the  swarm  provided 
reconnaissance  and  searched  for  improvised  explosive  devices  (IEDs)  on  and  around  the  path  of 
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the  convoy.  The  convoy  contained  four  trucks  that  followed  a  sequence  of  user-defined 
waypoints,  and  our  swann  consisted  of  40  ducted  fan  aircraft.  We  used  the  Unreal  Tournament 
2004  game  engine  to  conduct  our  simulation. 

We  used  a  potential  field  approach  for  swarm  control,  because  it  scales  easily  to  large 
heterogeneous  swarms  and  allows  a  user  to  dynamically  alter  swarm  behavior  by  adjusting  field 
parameters.  The  potential  field  used  is  a  nonlinear  sum  of  simpler  fields,  each  of  which  attracts 
or  repels  the  swarm  from  specific  battlefield  regions.  In  general,  there  is  a  field  that  controls  the 
swarm’s  primary  activity,  a  field  that  controls  obstacle  avoidance,  and  fields  that  control 
secondary  activities  (e.g.,  exploration  of  areas  of  interest).  Due  to  space  limitations,  we  will 
discuss  the  first  vector  field  in  depth  and  provide  a  short  description  of  the  potential  field 
approach,  which  is  further  discussed  in  Barnes  (7). 


The  swann’s  primary  activity  was  to  remain  in  close  proximity  to  the  convoy.  Suppose  at  some 
time,  t,  the  geometric  center  of  the  convoy  is  at  (xc,  yc)  and  that  all  the  convoy  vehicles  are 
contained  within  a  sequence  of  elliptical  rings  centered  at  (xc,  yc),  with  orientation  (j)  and  axis 
ratio  X.  Suppose  cj)  =  0,  these  rings  can  be  considered  as  contours  for  the  following  surface: 


Z(x,y)  =  e 


a((x-xc)2+hy-yc)2). 


(1) 


the  vector  field  will  be  derived  from  the  gradient  vectors  for  this  surface. 

To  keep  the  swann  near  the  convoy,  we  designed  a  control  scheme  to  attract  swarm  members  to 
a  specific  orbital  ring  of  the  contour  map  specified  by  the  inequality: 

(R*  -  ARm  )2  <  (x  -  xc)2  +  X(y  -  yc)2  <  (R*  +  AR0Ut  )2 , 

*  w 

with  R  >0,  AR-n  >  0  and  ARQut  >  0. 


Here  R  *  detennines  the  distance  between  the  desired  orbital  ring  and  center  of  the  surface  and 
ARout  +  ARm  determines  the  width  of  that  orbital  ring.  This  control  scheme  will  ensure  that  the 
swarm  will  remain  within  the  orbital  ring  described  in  equation  2. 


Suppose  (x,y)  is  a  point  outside  the  region  described  by  the  inequality  of  equation  2.  The  vector, 


vvvy 


wM  (x,y)~ 

^ out  (■ x,y ) 


tJ(x-xc)2  +X2(y-ycy 


(x-Xc) 

My-yc ) 


,  for  all  (x,y)*(xc,yc). 


(3) 


describes  a  vector  of  length  Win(x,y)  -  wout(x,y)  that  points  to  the  highest  point  on  the  surface  Z, 
(xc,  yc).  We  define  d(x,y)  to  be  a  weighted  Euclidian  distance: 

d(x,y)  =  yj(x-xc)2  +k2(y-yc)2  .  (4) 
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Let 


wout(x>y) 


_ 1 _ 

1  +  e  a  out  (d(x,y)-(R*  +AR 

out )) 


and 


Wm<Xy)  =  i- 


1 

l  +  Q^in  (d(x,y)-(R*  +ARin)) 


(5) 


(6) 


with 


aout 


In 


"out 


1  -  s 


and  a  in  =  V 


1  -  s 


(7) 


for  some  small  s  >  0.  The  functions  wout  and  Win  are  sigmoid  functions  that  approach  0  for  points 
within  the  elliptical  ring  defined  by  the  inequality  in  equation  2. 

Suppose  (Px,  Py)  is  a  point  of  interest,  such  as  the  location  of  a  potential  IED,  an  obstacle,  or 
another  vehicle.  The  vector 


w(x,y) 


(x-Px)^  +  (y-Pvr 


x-P.x 

vy“py; 


(8) 


can  attract  swarm  members  to  (Px,  Py)  if  w(x,y)  >  0  and  repel  swarm  members  from  (Px,  Py)  for 
w(x,y)  <  0.  In  our  work,  experimenter- assigned  attractors  were  used  to  direct  a  portion  of  the 
swarm  to  interesting  areas  on  the  battlefield,  which  we  designated  as  “hot  spots.”  As  we  stated 
previously,  the  final  vector  field  is  a  nonlinear  sum  of  vectors  defined  by  equations  3  and  8. 

To  introduce  metacognition  into  the  swann,  we  developed  metrics  that  the  swarm  could  use  to 
describe  its  state.  This  task  was  high  risk;  if  the  metrics  did  not  adequately  describe  the  state  of 
the  swarm,  the  swann  would  be  uncontrollable.  Our  swarm  metrics  included  the  number  of 
active  members  and  status  information  for  swarm  members  that  split  off  the  main  group.  We 
developed  operational  swarm  metrics  to  provide  infonnation  on  how  effectively  the 
metacognition  algorithms  performed.  One  operational  metric  we  developed  was  a  measure  of 
the  ability  of  the  swarm  members  to  maintain  an  orbital  ring  with  specific  size  limits  around  the 
entire  convoy,  while  the  convoy  traveled  over  a  tenain  with  experimenter-assigned  waypoints. 
Team  member  discussion  detennined  that  in  order  for  the  swann  to  provide  reasonable 
protection  to  the  convoy,  the  major  axis  of  the  swarm  orbital  ring  was  to  be  no  larger  than  two 
convoy  lengths  and  no  smaller  than  one  convoy  length.  We  conducted  an  experiment  to 
determine  what  proportion  of  time  the  simulated  swarm  was  able  to  maintain  the  previously 
described  orbital  ring  around  the  convoy  while  it  travelled  over  six  different  terrains  in  13-min 
scenarios.  Prior  to  the  experiment,  the  experimenter  assigned  waypoints  and  hot  spots  to 
different  terrain  locations.  During  the  experiment,  we  noted  what  proportion  of  time  the  swann 
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consistently  travelled  with  the  convoy  over  all  of  the  pre-defined  waypoints,  and  what  proportion 
of  time  the  swarm  members  were  attracted  to  the  preset  hotspots.  Success  in  this  area  would 
indicate  that  the  experimenter  had  successful  pre-simulation  control  of  swarm  actions.  Figure  1 
shows  a  swann  dispersed  around  a  convoy  on  a  simulation  terrain. 


Figure  1.  Convoy  and  swarm  on  simulated  terrain. 

To  design  the  Soldier-swarm  interface,  we  consulted  with  military  subject  matter  experts  and 
tested  swarm  behaviors  to  identify  different  types  of  information  that  the  swarm  might  present  to 
an  operator  with  swarm  oversight.  This  task  was  high  risk,  because  infonnation  incorrectly 
displayed  would  make  the  display  ineffective.  The  resulting  visual  swarm  display  is  shown  in 
figure  2.  Swarm  geospatial  status  showed  the  “live”  swarm  and  convoy  in  an  overhead  view,  as 
they  traversed  one  of  the  six  simulated  terrains  described  previously.  The  other  types  of 
information  (swarm  health,  swarm  communication,  and  convoy  status)  are  described  in  table  1 . 
We  added  display  information  in  different  modalities  (spatial,  audio,  and  tactile  cues),  because 
research  suggests  that  multimodal  cues  can  increase  awareness  of  surroundings  and  cue  visual 
attention,  especially  when  the  user  experiences  a  high  visual  load  (2).  Table  1  also  shows  our 
approach  for  presenting  information  in  different  modalities.  All  signals  were  triggered  by 
changes  in  swarm  health,  swarm  communication,  and  convoy  status. 
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Swarm  Display  Unit 


Geospatial  Map 


Swarm  Commo 


Swarm  Reformed 

Swarm  Health 

o  o  o  o 

Speed  Strength  Capability  Dispersion 


Figure  2.  Swarm  display. 

Table  1.  Information  presentation  by  display  type  for  swarm  display  elements. 


Swarm 

Display 

Visual 

(V) 

Visual  +  Audio 
(VA) 

Visual  +  Tactile 
(VT) 

Visual  +  Audio  +  Tactile 
(VAT) 

Swarm  health 

Status  lights  (green, 
yellow,  red)  for 
swarm  speed, 
strength,  capability, 
and  dispersion. 

Visual  cues  plus  recorded 
spoken  messages  (e.g., 
“Dispersion  Yellow”). 
Speech  (audio)  presented 
at  3 15°  from  center  front. 

Visual  cues  plus 
vibrating  tactor  at 
315°  from  center 
front. 

Visual  cues  plus  recorded 
spoken  messages 
presented  at  315°  from 
center  front. 

Swarm 

communications 

(commo) 

Text  box  shows  text 
messages  relating  to 
swarm  state  or 
activities  (e.g., 
“Swarm  Reformed”). 

Visual  cues  plus  recorded 
spoken  messages  (e.g., 
“Swarm  Reformed”) 
presented  at  0°  from 
center  front. 

Visual  cues  plus 
vibrating  tactor  at 

0°  from  center 
front. 

Visual  cues  plus  recorded 
spoken  messages 
presented  at  0°  from 
center  front. 

Convoy  status 

Text  box  at  lower 
right  shows  text 
messages  relating  to 
convoy  (e.g., 

“Convoy  Off  Road”). 

Visual  cues,  plus 
recorded  spoken 
messages  (e.g.,  “Convoy 
Off  Road”)  presented  at 
45°  from  center  front. 

Visual  cues,  plus 
vibrating  tactor  at 
45°  from  center 
front. 

Visual  cues,  plus  recorded 
speech  messages  at  45° 
from  center  front. 
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We  conducted  a  laboratory  study  to  evaluate  the  swarm  display  interface.  Sixteen  male  Marines 
with  a  mean  age  of  1 9  years  from  the  Marine  Detachment  at  Aberdeen  Proving  Ground,  MD, 
served  as  volunteer  participants.  Each  Marine  performed  the  task  of  monitoring  the  swarm 
display  and  responding  to  the  display  signals  when  they  occurred  by  pressing  one  of  three 
corresponding  buttons  on  a  pushbutton  display  and  then  verbally  telling  the  experimenter  which 
swarm  display  signal  they  recognized  (e.g.,  “Dispersion  Yellow”).  The  buttons  on  the 
pushbutton  display  corresponded  to  one  of  the  three  signal  types  (swann  health,  swarm 
communication,  convoy  status).  To  emulate  the  multitasking  environment  found  in  military 
missions,  the  Marines  simultaneously  perfonned  a  second  task.  In  the  second  task,  Marines  used 
a  modified  Robotic  Collaborative  Technology  Alliance  (CTA)  simulation  integration  laboratory 
(MSIL)  task  to  perform  four  different  cursor  control  tasks  (click  and  hold,  drag  and  drop,  point  to 
point,  and  text  selection)  that  resembled  robotic  planning  tasks  on  a  screen  resembling  the  SIL 
(shown  in  figure  3).  The  Marines  were  told  to  perform  both  tasks  as  quickly  accurately  as  they 
could,  but  to  respond  to  the  swarm  display  signals  as  soon  as  they  occurred.  Audio  cues  were 
played  over  Sony  headphones.  Tactile  cues  were  experienced  from  one  of  three  Engineering 
Acoustics  Incorporated  (EAI)  tactors  mounted  in  a  belt  worn  on  the  participant’s  torso. 


Figure  3.  MSIL  display. 
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The  independent  variables  were  swann  display  information  and  swarm  display  type.  Swarm 
display  information  was  swarm  health,  swann  communication,  and  convoy  communication. 
Swarm  display  types  were  visual  display  only  (V);  visual  display  supplemented  with  audio  cues 
(VA);  visual  display  supplemented  with  tactile  cues  (VT);  and  visual  display  supplemented  with 
audio  and  tactile  cues  (VAT).  The  dependent  measures  included  participant  response  times  to 
swarm  display  and  SIL  tasks.  The  National  Aeronautics  and  Space  Administration  (NASA) 
Task  Load  Index  (TLX)  rating  scale  was  used  to  evaluate  workload  (3).  A  final  questionnaire 
allowed  each  participant  to  compare  display  types  and  to  provide  additional  comments.  We  had 
two  experimental  hypotheses:  (1)  That  the  level  and  type  of  information  communicated  by  the 
display  would  furnish  adequate  infonnation  for  a  convoy  scenario,  and  (2)  that  visual  swarm 
displays  supplemented  by  audio  and/or  tactile  infonnation  (visual  +  audio,  visual  +  tactile,  and 
visual  +  audio  +  tactile  displays)  would  communicate  swarm  activities  to  the  Soldier  more 
quickly  and  accurately  than  the  swann  display  with  visual  infonnation  only. 


3.  Results 


The  results  of  the  metacognition  experiment  indicated  that  the  swarm  was  able  to  maintain  the 
pre-defined  dispersion  more  than  85%  of  the  time  on  each  of  the  six  simulation  terrains.  In 
addition,  the  experimenter  noted  that  the  swarm  consistently  travelled  with  the  convoy  over  all  of 
the  pre-defined  waypoints,  and  the  swann  members  were  attracted  to  all  of  the  preset  hotspots. 
This  indicated  that  the  experimenter  had  successful  pre-simulation  control  of  swarm  actions. 

Swarm  display  experiment  accuracy  data  indicated  that  99.9%  of  all  signals  were  conectly 
detected  and  recognized.  An  SPSS  mixed  model  analysis  of  variance  (ANOVA)  performed  on 
the  swarm  display  response  time  data  and  on  the  workload  data  indicated  significant  main  effects 
for  display  type  ip  =  0.000).  Bonferroni  post-hoc  tests  indicated  that  Marines  using  multimodal 
cues  (VA,  VT,  or  VAT)  had  a  significantly  shorter  response  time  and  lower  workload  than  those 
using  visual  cues  alone.  There  were  no  other  significant  differences.  Figures  4  and  5  show 
means  for  display  type  for  response  time  and  workload.  The  largest  mean  difference  in  response 
times  (between  V  and  VAT  displays)  was  1.5  s,  which  may  make  a  practical  difference  in  U.S. 
Army  operations  in  which  seconds  count.  Error  bars  for  both  figures  represent  95%  confidence 
intervals.  Marine  comments  indicated  that  their  preferences  were  consistent  with  the 
performance  and  workload  scores:  73%  preferred  the  VAT  display  over  other  display  types, 

20%  preferred  the  VA  display,  and  7%  preferred  the  VT  display.  The  V  display  was  least 
preferred  by  73%  of  the  Marines.  All  Marines  stated  that  they  liked  the  level  and  type  of 
information  content  of  the  displays.  Many  suggested  that  the  swarm  communication  display 
provide  additional  information  about  size  and  location  of  IEDs.  Suggestions  were  made  about 
swarm  formations  and  operational  concepts.  The  Marines  became  very  engaged  in  the  swann 
tasks  and  liked  the  idea  of  employing  swarms  in  future  military  operations. 
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Display  Type 


Figure  4.  Mean  swarm  response  time  for  each  display  type. 


Display  Type 


Figure  5.  Mean  NASA  TLX  workload  rating  for  each  swarm  display  type. 
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4.  Conclusions 


Most  of  the  first-year  objectives  of  this  two-year  DRI  research  plan  were  met.  We  successfully 
defined  a  40-member  simulated  swarm  to  accompany  a  four-member  convoy,  and  successfully 
developed  metacognition  algorithms  that  enabled  swarm  members  to  efficiently  monitor 
changes  in  swarm  status  as  they  executed  six  different  convoy  missions.  We  determined  that 
the  experimenter  could  successfully  assign  waypoints  and  hot  spots,  indicating  that  pre¬ 
simulation  control  of  swarm  actions  was  possible.  We  successfully  designed  a  human-swarm 
interface  that  allowed  Marines  to  efficiently  interact  with  a  robotic  swarm  participating  in  a 
representative  convoy  mission.  We  succeeded  in  providing  swarm  display  infonnation  concepts 
that  could  efficiently  and  effectively  communicate  swarm-provided  information  to  Marines 
by  defining  specific  information  demanded  for  a  Soldier  swann  supervisor.  The  Marines  in 
our  experiment  indicated  that  the  information  provided  by  the  interface  permitted  them  to 
interact  with  the  swarm  as  one  homogeneous  entity,  rather  than  an  entity  with  40  different 
members.  We  were  also  successful  in  developing  different  multimodal  display  concepts  for 
the  human-display  interface. 

Two  of  our  DRI  goals  were  not  completely  met: 

1 .  We  still  need  to  further  define  non-operational  criteria  for  swarm  metacognition. 

2.  Swarm  control  by  a  human  during  swarm  operations  has  not  yet  been  accomplished. 

These  high-risk  tasks  will  be  attempted  in  the  second  year  of  the  DRI,  which  will  deal  with 
expanded  swann  strategies  for  more  complex  swarm  operations  and  with  real-time  human 
interfaces  for  swarm  control.  Elements  of  our  completed  research  (i.e.,  our  observations 
regarding  the  limitations  of  our  current  metacognition  algorithms  and  Marine  suggestions 
regarding  relevant  swann  control  tasks)  will  serve  as  a  basis  from  which  to  approach  these 
second-year  tasks. 
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6.  Transitions 


Our  work  will  be  used  to  support  our  year  two  DRI  research  in  which  Soldiers  will  be  able 
to  modify  the  behavior  of  the  swarm.  It  will  also  be  used  for  studies  of  heterogeneous  swanns 
(ground  vehicles  and  helicopters)  in  future  U.S.  Army  Research  Laboratory  (ARL)  and 
University  of  Texas  in  Arlington  research.  In  addition,  the  Micro  Autonomous  Systems 
and  Technology  (MAST)  CTA  has  shown  interest  in  the  algorithms  developed  for 
the  simulated  swann. 

A  technical  report  is  being  prepared,  as  well  as  papers  for  submission  at  conferences,  including 
the  Human  Factors  and  Ergonomics  Society  and  the  Association  for  Computing  Machinery 
(ACM)/Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  Human  Robot 
Interaction  conferences. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ACM 

CTA 

DRI 

EAI 

HRI 

IEDs 

IEEE 

MAST 

MSIL 

NASA 

SIL 

TLX 

V 

VA 

VAT 

VT 


Association  for  Computing  Machinery 

Collaborative  Technology  Alliance 

Director’s  Research  Initiative 

Engineering  Acoustics  Incorporated 

human-robot  interaction 

improvised  explosive  devices 

Institute  of  Electrical  and  Electronics  Engineers 

Micro  Autonomous  Systems  and  Technology 

modified  simulation  integration  laboratory 

National  Aeronautics  and  Space  Administration 

simulation  integration  laboratory 

Task  Load  Index 

visual  display  only 

visual  display  supplemented  with  audio  cues 

visual  display  supplemented  with  audio  and  tactile  cues 

visual  display  supplemented  with  tactile  cues 
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